.\"	$NetBSD: boost-test-suite.1,v 1.6 2009/03/06 23:48:42 brook1 Exp $
.\"
.\" Copyright (c) 2009 Brook Milligan.  < brook -at- nmsu.edu >
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
.\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd February 7, 2009
.Dt BOOST-TEST-SUITE 1
.Os
.Sh NAME
.Nm boost-test-suite
.Nd run Boost test suite
.Sh SYNOPSIS
.Nm
.Op Fl Vcdhipv
.Op Fl D Ar domain
.Op Fl I Ar URL
.Op Fl N Ar name
.Op Fl S Ar schedule
.Op Fl T Ar title
.Op Fl U Ar username
.Op Fl b Ar directory
.Op Fl l Ar logfile
.Op Fl t Ar toolsets
.Op -- Ar args
.Sh DESCRIPTION
The
.Nm
program provides a simple framework for running tests of Boost
(http://www.boost.org/).  As it runs,
.Nm
takes care of downloading the necessary Boost test script and running
it with parameters appropriate for simple testing applications.  It
may, however, be configured for more advanced uses, as the options
.Ar args
are passed directly to the test script.
.Pp
.Nm
also generates a file,
.Dq comment.html ,
that is required by the Boost test suite for reporting the test
results.  This file contains information summarizing the test
environment, including how to retrieve the patches used in the test.
A number of command line arguments serve to control the content of
this file.
.Pp
The testing process and requirements are described in greater detail
at the following site:
.Pp
   http://www.boost.org/development/running_regression_tests.html
.Pp
Note that the Boost test framework requires a network connection, as
it updates the local copy of Boost with the most current version prior
to running the tests and then uploads the results, which are
summarized here:
.Pp
   http://www.boost.org/development/tests/trunk/developer/summary.html
.Pp
Any patches used in the test are imported either directly from the
installed package, which comes with the current set of patches, or
from a URL referring to an
.Dq index file
describing the patch set (See
.Sx INDEX FILE FORMAT
for more information).
.Pp
In general use, it is best to run tests both with and without patches
(i.e., without and with the
.Fl p
option) on the same machine.  This provides the most useful
information for both the Boost the pkgsrc communities.  Further,
unless you intend to develop new sets of patches, it is best to rely
on the standard set of patches imported by default (i.e., without
specifying either the
.Fl I
or the
.Fl i
options).  This will ensure consistency among instances of the tests
so that a wide array of platforms will be tested with the same
configuration.
.Pp
It is important that the pkgsrc community run the Boost test suite
often as it not only improves cross-platform support for Boost itself,
but it improves our packages by regularly testing our pkgsrc-specific
patches and improves the quality of our compilers by exercising the
toolchain.
.Pp
Please contribute available resources to the testing so we can ensure
the best Boost packages and compilers possible.
.Pp
.Sh OPTIONS
The following options are available:
.Bl -tag -width indent
.It Fl D Ar domain
Insert
.Ar domain
into the
.Dq comment.html
file as the domain portion of the email address for contacting the
operator responsible for the test suite runs.  The structure of the
.Dq comment.html
file tries to avoid creating full email addresses from this
information that can be directly harvested.
.It Fl I Ar URL
Import the index file from
.Ar URL ,
which must be interpretable as a URL by
.Xr ftp 1 .
By default, patches are imported from a standard repository to
maintain consistency among test instances.  However, nonstandard index
files may be made available, thereby allowing the testing of
additional patch sets.  See
.Sx INDEX FILE FORMAT
below for a description of such index files.
.It Fl N Ar name
Insert
.Ar name
into the
.Dq comment.html
file as the name of the operator responsible for the test suite runs.
.It Fl S Ar schedule
Insert
.Ar schedule
into the
.Dq comment.html
file as the comment describing the scheduling of test suite runs.
.It Fl T Ar title
Insert
.Ar title
(together with basic system information) into the
.Dq comment.html
file as the title of the test suite runs.
.It Fl U Ar username
Insert
.Ar username
into the
.Dq comment.html
file as the user portion of the email address for contacting the
operator responsible for the test suite runs.  The structure of the
.Dq comment.html
file tries to avoid creating full email addresses from this
information that can be directly harvested.
.It Fl V
Print version information and exit.
.It Fl b Ar directory
Run the Boost tests in
.Ar directory .
By default, the tests will be run in @BOOST_TEST_BASEDIR@.
.It Fl c
Clean and remove the test directory after running the tests.  If the
tests fail, this cleanup will not occur so that the files can be
examined to determine the cause of the failure.
.It Fl d
Generate debug output and include it in
.Dq comment.html .
.It Fl h
Print a brief help page.
.It Fl i
Do not import patches.  Instead, use the fixed set distributed with
the package.
.It Fl l Ar logfile
Log the output of the test runs to
.Ar logfile .
Note that a relative path will be resolved relative to the test
directory.
.It Fl p
Do not patch the source prior to running the tests.
.It Fl t Ar toolsets
Use
.Ar toolsets
(a comma-delimited list of valid toolsets) as the
.Dq toolsets
argument to the Boost test script.
.It Fl v
Increase the verbosity of output.  This option may be repeated.
.El
.Sh INDEX FILE FORMAT
Sets of patches are defined in an
.Dq index file .
The URL of this file is specified with the
.Fl I
command line option; if this is not given, a standard URL is used
instead.  The purpose of this file is to define both the URL used to
refer to patches in the
.Dq comment.html
file and the URL(s) used by
.Xr ftp 1
to download the patches.  Blank lines and comments beginning with #
are ignored.  A single line beginning with `URL:' defines the URL
inserted into
.Dq comment.html .
All other lines are expected to be valid
URLs, interpretable by
.Xr ftp 1 ,
specifying patch files to import.
.Sh EXAMPLES
.Nm
can be usefully run with no command line options or with only the
.Fl p
option (to avoid patches altogether).  However, because the test
results will be embedded within a large table of other results, it is
best to specify some identifying information that is more useful than
the defaults.  The following command gives the test an explicit title
(Drofnats) and provides contact information so that inquiries about
the test can be directed to the right person (in this case
R.J. Drofnats, rj@drofnats.com).
.Pp
boost-test-suite -T Drofnats -N "R.J. Drofnats" -U rj -D drofnats.com
.Pp
Ideally, the Boost test suite will be run on a regular basis with and
without patches.  This may be accommplished with the following
.Xr crontab 5
entries.
.Pp
#minute hour    mday    month   wday    command
.br
0       2       *       *       *       @PREFIX@/sbin/boost-test-suite
.br
0       14      *       *       *       @PREFIX@/sbin/boost-test-suite -p
.Pp
In these cases it is most apprpriate to include additional identifying
information (such as shown in the first example above).  Further, if
the tests are run regularly, it is appropriate to use the
.Fl S
command line option to indicate the schedule.
.Pp
boost-test-suite -S "Tests are run regularly at 0200 UTC-8"
.Pp
Of course, in practice all of the options illustrated here should be
combined to construct an appropriate set of command line options.
.Sh SEE ALSO
.Xr cron 8 ,
.Xr crontab 5 ,
and
.Xr ftp 1 .
.Pp
More information about Boost and the testing process is avaiable from
http://www.boost.org/ and
http://www.boost.org/development/running_regression_tests.html.  The
test results themselves are available from
http://www.boost.org/development/tests/trunk/developer/summary.html.
.Sh AUTHOR
Brook Milligan
